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Abstract— This paper describes a structured light-based sen- 
sor for hazard avoidance in planetary environments. The system 
presented here can also be used in terrestrial applications 
constrained by reduced onboard power and computational 
complexity and low illumination conditions. The sensor consists 
on a calibrated camera and laser dot projector system. The 
onboard hazard avoidance system determines the position of 
the projected dots in the image and through a triangulation 
process detects potential hazards. The paper presents the design 
parameters for this sensor and describes the image based 
solution for hazard avoidance. The system presented here was 
tested extensively in day and night conditions in Lunar analogue 
environments. The current system achieves over 97% detection 
rate with 1.7% false alarms over 2000 images. 


I. INTRODUCTION 


Planetary rover navigation is constrained by a number of 
factors including real-time operation requirements, limited 
onboard computational resources and power, and lack of de- 
tailed knowledge of local terrain and illumination conditions. 
Operations on airless bodies, such as the Moon, Mercury 
and asteroids, are additionally notable for their extreme 
imaging conditions of high dynamic range (bright sunlight, 
dark cast shadows) and backscattering regolith [1]. However, 
these bodies are also among NASA’s highest priority science 
destinations for sample return and volatiles confirmation.[2]. 

Autonomous and semi-autonomous rover capabilities are 
still nascent for airless bodies, particularly in the area 
of obstacle avoidance. The work presented in this paper 
describes a structured light ’virtual bumper” system for 
obstacle avoidance in these environments. Virtual bumpers 
are non-contact, in-close safeguarding sensors that are the 
last line of defense for hazard detection. Furthermore, due to 
power considerations, they are typically the only always-on 
navigation sensors while driving in regions of total darkness 
(e.g., craters and caves). As such, they must detect obstacles 
reliably and in real-time within the restrictive requirements 
of planetary exploration missions. 

The virtual bumper system we have developed has been 
tested in an outdoor test facility under a variety of illumi- 
nation conditions including bright daylight and night time 
darkness. While designed specifically for planetary envi- 
ronments, the hazard avoidance system presented here has 
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terrestrial rover applications including mine and cave explo- 
ration and night time traversing. The system uses one or more 
calibrated laser beam dot projectors and a camera. System 
design parameters include laser beam intensity, wavelength, 
orientation and inter-beam angles. A low complexity image 
processing algorithm detects the projected dots, and based on 
their location in the image, determines the size of obstacles 
in the rover’s path. 

The hazard avoidance system design is described in Sec- 
tion III. The image-based onboard hazard detection algorithm 
is described in Section IV, and the experimental results are 
described in section VI. 


II. PRIOR WORK 


Virtual bumper systems have long been employed on 
mobile robots. Early terrestrial applications utilized non- 
contact sensing to detect intrusion of personal space in 
autonomous highway control strategies [3], [4] utilizing 
RADAR. In modern field robotics, LIDAR virtual bumpers 
are the preferred method of in-close obstacle detection [5]. 
Space robotics, in contrast, has not readily adopted time- 
of-flight technology. Instead, much focus has been placed 
on structured light triangulation for rovers, and such sen- 
sors have been conceived for hazard detection in planetary 
missions [6] like Mars Pathfinder [7], [8] and for smallsats 
[9]. 

Structured light is particularly advantageous in space 
application due to leveraging of existing cameras and the 
simplicity of laser diode/diffraction grating hardware. Unlike 
LIDAR and RADAR, cameras have long legacy of space 
qualification (radiation, thermal, vacuum) and benefits of low 
mass and power. Use of active illumination, in addition, has 
advantages over passive stereo in extreme planetary illumi- 
nation conditions. The Mars exploration rover, Opportunity, 
was unable to get acceptable stereo range information from 
hazard cameras due to lack of texture in the Mars terrain 
[10]. In dark environments, which are the particular focus of 
this paper, stereo cannot work without power hungry, active 
area illumination and is susceptible to dynamic range issues 
present in imagery from airless bodies. 

Specific use of virtual bumpers as a part of a tele-operated 
robotic system in dark environments was explored in [11]. 
Three-dimensional, commercial structured light sensing, in 
the form of Kinect, was utilized in [12] for capability in 
dark terrestrial rescue environments, but is noted for its 
poor performance in the presence of direct sunlight. Two- 
dimensional light stripers have been explored for operations 


in direct sunlight, but not with considerations relevant to the 
Moon [13], [6]. This paper extends prior work to consider 
structured light virtual bumper technology for lunar surface 
illumination conditions which straddle the extremes of total 
darkness and harsh direct sunlight. 


III. STRUCTURED LIGHT DESIGN 


The dots projected by the system must be visible under 
all ambient illumination conditions, including both bright 
sunlight and total darkness. The conditions on the Moon 
are considered for this analysis, particularly near the lunar 
poles to which several potential missions are planned [2]. In 
this scenario, both brightly sunlit and extremely dark regions 
shadowed by crater walls or the day-night boundary may 
appear in the same image. 

One operational mode is to repeatedly capture two images 
from the same location while modulating the projector, one 
showing the projected dots and the other taken without them. 
Then, a simple frame differencing-based method can be used 
to determine the location of the projected dots. However, 
such a solution, while feasible under many other conditions, 
would require that the rover stop, which fails to satisfy 
the requirement of real-time operation. Instead, the method 
proposed in this paper detects the laser dots from a single 
image and requires simply that the projected dots outshine 
all other illumination. This is accomplished by ensuring that 
the ratio between the pixel intensity due to the laser projector 
illumination (Liaser) and the pixel intensity due to ambient 
illumination (Ls) be larger than a fixed threshold T. 
The system design parameterized by laser projector power, 
wavelength, and operational distance is described by 
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where Praser is the projected power over each dot on the 
ground, 2 is the laser wavelength, and r is the slant range 
calculated as the distance between the laser projector and the 
intersection between the laser beam and surface (Figure 7). 

As a simplification of the full light transfer function [14], 
the pixel intensity due to laser and Sun illumination is 
modeled by Equations 2 and 3 respectively, 
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where A(A) is the lunar albedo at a given wavelength, 
R(a,i,e) is the Lunar-Lambertian reflectance model chosen 
here, [,(A) is the wavelength-dependent Sun irradiance, S'(r) 
is the surface area on the ground seen by a pixel in the image 
at a slant range r from the camera, C(A) is the camera 
response as a function of the wavelength », and F(A) is 
wavelength-dependent attenuation due to an optional optical 
filter placed on the camera. 

Figure | illustrates the lunar albedo [1] for several lu- 
nar terrain regions and a wavelength range between 400- 
3000nm. The lunar reflectance model [17] is given by 
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Fig. 1. Albedo for various lunar terrain regions, using soil samples 


12001, 14163, and 67641 from [15] and [16]. 


Fig. 2. Reflection angles used in the Lunar-Lambertian model are measured 
between the view vector (v), lighting vector (1) and the surface normal (n). 


Equation 4: 
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cos(i) + cos(e) 
where 7, e and « are the incidence, emission and phase angles 
illustrated in Figure III respectively. The incidence angle is 
defined as the angle between the vector of the light source 
and the surface normal vector at a point on the surface. The 
emission angle is defined as the angle between normal vector 
and vector of the camera at the same point on the surface. 
The angle between the light source vector and camera vector 
at a point on the surface denotes the phase angle. 

The solar irradiance as a function of the wavelength is 
shown in Figure 3. In a terrestrial application, several atmo- 
spheric absorption bands greatly reduce the sun’s intensity at 
many wavelengths, however in the case of the moon lacking 
any substantial atmosphere, solar irradiance may instead be 
modeled as originating from a pure blackbody radiator. 

Figure 4 illustrates an example camera response function 
C(A) as a function of the wavelength. An important addition 
to reduce ambient illumination is the use of an optical filter 
modeled by F'(A) that passes a desirable wavelength band 
(selected to be centered around the laser projector wavelength 
(Equation 2) and filters out most of the Sun illumination 
(Equation 3). For typical filters, F(A) may be modeled 


F 
C) 


—ldeal blackbody irradiance (Moon) 
—— Solar irradiance on Earth surface 


Be ies ee 


Spectral Irradiance [W/m2/nm] 
go eg-9¢ 9: 
N B © © Ff 


oO 
300 400 500 600 700 800 900 1000 
Wavelength [nm] 


Fig. 3. Solar irradiance on the surface of the earth (with an absorptive 
atmosphere) and moon (without one). [18] 
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Fig. 4. Camera transfer function for a typical monochrome imager, the 
Sony ICX267 from the Allied Vision Manta machine vision camera. [19] 


as roughly a Gaussian-shaped pass band with a width of 
approximately 100nm. 

The surface on the ground covered by a pixel in the image, 
S is approximated by Equation 5, 


(5) 


where f is the camera focal length in pixels. 
Note from Equations 5 and 1 that the shorter the distance 
r the lower becomes the threshold 7 making the sensor 
more robust and more power efficient. However, the distance 
on the ground between the rover wheels and the obstacle 
must be larger than the distance required by rover to stop 
dstop if an obstacle is detected. The distance dstop) is the 
distance traveled by the rover during image acquisition, 
processing and before comes to a complete stop. In a typical 
configuration the laser projector is oriented at a pitch angle 
@ from the rover body as shown in Figure 7. The following 
equation 
rtan@ > dstop (6) 


gives a lower bound on the slant range. Equation 6 also 
shows the importance of a reduced image acquisition and 
processing time. This constraint plays an important role in 
the design of the hazard detection algorithm described in 
Section IV. 


A second aspect of the structured light sensor design is to 
determine its parameters that allow for detection of obstacles 
of a given height and in a given range w perpendicular to 
the direction of traverse. These parameters include the pitch 
angle @ of the sensor with respect to the rover (Figure 7), 
the number of laser beams NV, and inter-beam angle y as 
illustrated in Figure 5. The laser projector and camera are 
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Fig. 5. N laser beams separated by inter-beam angle +. 
mechanically aligned such that they both have the same roll, 
pitch, yaw and are separated only by a horizontal baseline, b. 
For a general surface illuminated by the laser projector, the 
dot positions in the image change compared to the expected 
dot position on a flat surface. This pixel disparity dd relates 
to the slant range r between the camera and surface along a 
camera ray as described in the equation below 


2 
or = —dd 7 
es (7) 
where f is the camera focal length (in meters). The trian- 
gulation process is illustrated in Figure 6. If the camera and 
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Fig. 6. The projector and camera triangulation process. Obstacles at 
distance r and r — 6r appear in the image separated by a pixel disparity 
éd. 


laser projector system are looking at the surface under a pitch 
angle 0, (Figure 7) the variation in obstacle height dh that 
corresponds to a pixel disparity dd is given by 
2 
dh = ér cos(@) = 7a cos(@) (8) 


The above equation defines the minimum pixel disparity 
to detect obstacle higher than a given dh. The number of 
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Distance from laser projector mounted on the rover to ground 


laser beams N to cover a segment w perpendicular to the 
rover traverse at a distance rsin(@) in front of the rover is 
calculated as 


= |ole 


2 
N =—tan-*(4)+1 (9) 
Y 
If the number of laser beams cannot be achieved with single 
laser projector, a solution with multiple projectors is chosen 
to satisfy Equation 9. The number of beams must also satisfy 
the condition 
w 
— > dd 
N > 
If the above condition is not satisfied the system will detect 
as obstacles terrain features that are smaller than dh and 
generate a large number of false alarms. 


(10) 


IV. HAZARD DETECTION 


The hazard avoidance system uses the laser dots pro- 
jected in the camera to determine potential obstacles in the 
rover path. In a calibrated system the camera and laser 
dot projector have the same orientation and pose except a 
horizontal baseline b as described in Section IIL. For this 
system all projected dots align on the same epipolar line. 
The obstacles in front of the camera will determine only 
a shift in the horizontal position of the laser illuminated 
pixels. Their vertical position remains on the same image 
row. This solution reduces significantly the image acquisition 
and processing time since the search for the laser dot location 
reduces to a single image line. It also relaxes the condition 
in Equation 6 by reducing the distance dst.) needed by the 
rover to stop in front of an obstacle. Let ROW_INDEX be 
the image row where all laser dots are projected and let 
LEFT_COL and RIGHT-_COL be the columns corresponding 
to the left most and right most laser beams projected on 
the image on a flat surface. An image region of interest 
(ROT) is defined by the following boundaries. The top left 
corner of the ROI is (LEFT_COL-dd, ROW_INDEX-D) and 
right bottom corner is (RIGHT _COL+dd, ROW_INDEX+D), 
where D is a fixed number of image rows. dd is computed 
using Equation 8 for the given minimum obstacle height 6h, 
and given sensor pitch 9. Note that with this choice of 6d and 
ROI boundaries and in the presence of obstacles of height 
larger than dh, the number of dots projected within the ROI 
is lower than the number of laser beams NV. 


Figure 8 describes the overall system for hazard avoidance 
using the structured light sensor. 


Fig. 8. Overall system 


Reference Subtraction and Thresholding 
When there are two images available captured with and 
without the laser projected dots from the same camera 
pose and same illumination conditions, the image difference 
between these two images will select primarily the pixels 
illuminated by the laser together with low intensity image 
noise. This solution requires the rover to stop and insure 
the two images are captured from the same camera pose. 
The threshold value is selected such that pa > T. 
Using a single image showing the projected laser dots 
allows to select all pixel illuminated by the laser projector 
using a fixed threshold value T such that Fase: > T. In 
this case the laser projected power must be significantly 
higher to insure that terrain features of various slopes and 
albedo values are not confused for projected laser beams 
(Equation 1). Alternatively, under the assumptions of slow 
varying terrain features, a *background” ROI can be used. 
The background ROI has the same width, height and left 
horizontal pixel value as the ROI containing the projected 
laser dots. It is only shifted vertically by a fixed number of 
pixels. The ’background”ROI approximates the appearance 
of the background image when the background image is not 
available and the threshold value T' in this case is selected 
such that piaser. > T. This solution allows for continuous 
rover operation and for reduced laser power intensity but 
relies on the assumption of a slow varying terrain features. 
The output of this processing stage is a binary image of the 
ROI size obtained through thresholding. Pixels with intensity 
values above T' value are set to one. All other pixels are set 
to zero and discarded. 

Clustering and Filtering by Size 
The pixels in the binary image are grouped into several clus- 
ters using a connected component algorithm. The centroid of 
each cluster is computed through averaging of the position of 
all pixels belonging the the cluster. Clusters with a number 
of pixels below a fixed threshold are removed. 

Obstacle Decision 
The detection algorithm determines the number of laser dots 
within the ROI and their relative position. When the number 


of detected centroids in the ROI is lower than the number 
of laser beams NV, the algorithm detects an obstacle. This is 
due to the size of the image ROI relative to the maximum 
pixel disparity for a given acceptable obstacle. This case 
corresponds to occlusions or the detections of wide obstacles 
(positive or negative) or terrain slopes that shift all laser dots 
horizontally by more than dd. 

If the number of detected dots equals the number of 
laser beams N, a simple correspondence based on pixel 
order is established between the expected location of the 
dots on a flat surface and their detected location. If the 
(horizontal) distance between the expected and detected dot 
position is larger than dd for any dot, the algorithm detects an 
obstacle. This case corresponds to narrow obstacles (positives 
or negatives) that do not influence the location of all dots but 
only a subset of them. 


V. EXPERIMENTAL SETUP 


A virtual bumper prototype was developed for testing 
and is shown in Figure 9 (right). The system utilizes a 
commercial machine vision camera [19] and a commercially 
available red light (660nm) laser projector with 11 dot 
diffraction grating [20]. The baseline between camera and 
laser projector is set to 26 cm and the pitch angle 0 = 28 deg. 

Physical experimentation was performed with the virtual 
bumper on NASAs K-REX2 rover (Figure 9). The goals 
were to stress the design and learn about the parameters 
and performance in a Lunar-relevant environment. Testing 
occurred at NASA Ames Research Centerss Roverscape 
facility, a two-acre outdoor planetary analog terrain with 
boulder distributions and average surface albedo (8%) similar 
to Lunar regolith. However, the free surface is covered 
in a layer of pea gravel and not a regolith (moon dust) 
simulant for reasons of practicality. Day and night tests were 
conducted to recreate the illumination conditions in direct 
sunlight and inside shadowed craters, respectively. Though 
daytime illumination distribution and spectrum are not exact 
standins for the moon, which lacks an atmosphere, clear sky 
days were leveraged such that total incident illumination and 
dominant directionality were similar. 


Fig. 9. 
during night testing in the Roverscape (left) . The virtual bumper experi- 
mental setup comprises a multi-dot laser projector and camera mounted on 
the leading edge of the rover (right) 


Experimentation utilized NASAs K-REX2 rover, pictured here 


Obstacle test cases were designed to mirror the geometric 
possibilities likely to be seen in craters at the poles of the 
moon. Figure 10 shows examples of the four main types of 


the obstacles encountered along with their relative sizes and 
reflectivities: large angular rocks characteristic of the polar 
surface areas, ejecta blocks characteristic of craters, smaller 
angular rocks, and complex aggregates from multiple small 
obstacles. Obstacle albedos ranged from <10% to 33%, and 
were dissimilar to the general terrain surface albedo. This 
presents a way of testing incident faces on obstacles that 
would be free of regolith accumulation. Within the scope 
of this paper, all of the obstacles tested are positive (rocks) 
which reflects the relative prevalence of rocks versus craters 
and gaps (negative) in the size danger zone. 

The K-REX2 rover was manually driven in the Rover- 
scape while pushbrooming the virtual bumper out front. 
Contact with obstacles was always made in the direction 
of pushbrooming that is, no point turn obstacles. A variety 
of speeds between 0 and 20cm/s were tested with 10cm/s 
being the most common. Manual driving enabled the rover 
to weave cleanly between and reverse from obstacles without 
biasing results with navigation software in the loop. Images 
were taken at 0.5Hz rate, reflecting a mission requirement, 
timestamped, and processed offline. Obstacles were hand 
labeled in images with guidance from flagged timestamps 
where obstacles were encountered. The camera was locked 


Fig. 10. Rock types used in day and night testing included (left to right) 
large angular rocks, ejecta blocks, small angular rocks, and complex multi- 
rock aggregates. 


at F/5.6 aperture, focal length 3.5mm, and ISO400 equivalent 
gain, while independent shutter speeds were chosen for day 
and night testing. These fixed shutter values were hand-tuned 
to produce 90% saturation of the laser dots in the resulting 
image (i.e. 0.9x255 = 230 for 8-bit images). Ideally, a 
single shutter speed would be used for all conditions, thus 
enabling use of the system across bright scenes with dark cast 
shadows. However, due to dynamic range limitations of the 
commercial machine vision camera, it was decided to collect 
clean data instead of introducing a minor operational detail. 
Other optical settings were chosen to reflect good depth of 
field and low pixel noises. 

An image threshold controls what brightness level to 
accept if a pixel is illuminated by a laser dot (Figure 11). 
Two optimal thresholds exist for night and day due to 
the differences in total exposure. The large spread between 
these empirical values ( 50%) stems from both the decision 
to use different shutter speeds and the change in ambient 
illumination. We believe that while two optimal thresholds 
will exist due to the latter reason, the much smaller spread 


will allow a single threshold to work well in night and day 
images. 

The minimum pixel disparity threshold dd determines if 
each laser dot has detected an obstacle based on the distance 
between the dot and its ideal flat-groundplane location along 
the epipolar line as discussed in Equation 8. The analytical 
value of dd for our system parameter and obstacles of 
height h = 10cm is 15.9 pixels. As the system is designed 
with a neutral spacing of 20 pixels between laser dots, 
there is an upper disparity limit at which it is impossible 
to identify a shifted dot between neighbors (and granular 
obstacle detection capability is lost), though the software 
allows for such values (Equation 10). Based on validation 
data, it was empirically determined that a threshold of 15 
(75% of the ideal spacing) strikes the optimal balance of 
precision and sensitivity (Figure 12). 


Binary Image Threshold Validation 
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Fig. 11. Selection of binary image threshold values using Fl score as a 
metric. Spread of data between day and night are due to natural changes in 
ambient illumination and artifical changes in camera settings. 


Disparity Threshold Validation 
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Fig. 12. Validation of the minimum disparity threshold parameter for 


detecting obstacles on experimental data. The value found empirically (15 
pixels) is close to the analytic ideal of 15.9 pixels. 


VI. RESULTS AND DISCUSSION 


Results from testing the virtual bumper on the Roverscape 
are shown in Figure 13. 1158 day images across two tests 


and 1544 night images were used in evaluation of the virtual 
bumper. Between 30-40% of these images were of obstacles 
and the rest were from clear paths. Two daylight tests were 
run with mechanical configurations that optimized detection 
for different target distances. Tests utilizing the second, 0.7m 
configuration were conducted when it was noticed that the 
apparent brightness of the laser dots over the background ter- 
rain was much less than anticipated (see discussion of errors 
below). The closer target distance increased the perceived 
brightness of the laser dots due to the reduction in square 
distance illumination falloff and a change in incident angle 
on the terrain towards the perpendicular. 

The positive obstacle detection accuracy of virtual bumper 
is around 97%, and is not significantly different between 
the day and night images. However, the false alarm rate 
(detection of obstacle when actually clear) varies signif- 
icantly from 1.7% at night to 4.1% in the worst of the 
day tests. This concurred with the visual observation that 
dots were challenging to perceive in daylight, despite use of 
monochromatic laser and narrow band filter. The average Fl 
score for day images is 0.92 compared to 0.96 for night. 


Day | Night 
Target Distance 0.7m 12m 12m 
Total Images (#) 529 629 1544 
# Obstacles 119 147 444 
# Clear Path 410 482 1100 
| Ce | 
True Pos 117 139 431 
True Neg! 395 462 1081 
False Pos 15 20 19 
False Neg' 2 8 13 
Obstacle Acc (%) 98 95 97 
False Alarm Rate (%) 3.6 41 1.7 
Fl Score 0.93 0.91 0.96 
Total Day Images 1158 
Avg Day Fl 0.92 
Fig. 13. Table of Results 


Images representing typical erroneous classification in day 
and night testing are shown in Figure 14. In day and night 
images, the most common false positive errors were the 
dots speckling as the rover drove over clear terrain or the 
introduction of new dots in the region of interest by bright 
terrain spots (day only). The major source of these errors is 
the result of terrestrial factors not represented in the ideal 
reflectance model. These factors included the blanket of 
granular pea gravel, compared to fine, powdered regolith 
which covers the moon. These granules both act as tiny 
occluders and introduce variegation in the terrain as they 
are not made of a macroscopically uniform material. The 
total effect was such that the variability in surface terrain 
reflectance was at times greater than the contribution of the 
laser illumination, causing dots to speckle in the imagery. In 


day and night images, many false negative errors stemmed 
from labeling at the transition between clear ground terrain 
and rock obstacles. These labels have great uncertainty 
and given possibilities to exploit the sequential nature of 
pushbroom images, great performance in these regions is 
not important. Upon discarding these transition images, the 
positive obstacle accuracy is over 99%. A source of false 
negative error specific to night images is smearing of dots 
due to motion blur from long exposure time. While a dot 
that has moved sufficiently along the epipolar line should 
trigger a detection, motion blur creates an elongated blob 
where the centroid is within the disparity tolerance. This has 
reinforced the importance of fast shutter speeds, though it 
was unavoidable at the time due to camera availability. 

It can be argued that the most important purpose of the 
system to is correctly identifying obstacles which would 
cause catastrophic failure. In this, virtual bumper has per- 
formed quite well. Moreover, virtual bumper is perhaps the 
last line of defense in dark craters, where a Lunar rover will 
be spending most of its time with intermittent stereo and flash 
illumination. The strong dark performance demonstrated here 
highlights virtual bumpers value for this niche purpose. 
Thus, these results are promising for early proof-of-concept. 
However, in future development, more realistic reflectance 
environments will be emphasized and with laser technology 
with greater power output will be investigated. 


False Positive False Negative 
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Fig. 14. Error examples from day and night testing. False positives result 
from variegation of terrain. False negatives result from uncertain labeling 
of transition regions between obstacle and clear ground. Motion blur also 
caused false negatives in night images. 


VII. CONCLUSIONS 


This paper demonstrates the usability of a structured 
light sensor for hazard avoidance in planetary navigation as 
well as for terrestrial applications restricted by low onboard 
power and low computational complexity. The sensor design 
methodology for a given set of constraints is discussed and 
a low complexity image-based hazard avoidance method 
is presented. The method has been tested in several day 
and night scenarios in a Lunar analog environment. The 
current system achieves over 97% detection rate with 1.7% 


false alarms. Future work will be directed towards the use 
of higher power laser projectors. The capabilities of the 
image based hazard avoidance method will be enhanced to 
handle robustly high reflectance surface regions and multiple 
reflections from the same projected beam. 
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